Remote gaming applications, in which a server-side game is controlled by a client-side player, have attempted to encode the video output from a three-dimensional (3D) graphics engine in real-time using existing or customized codecs, also referred to as encoders. However, the interactive nature of video games, particularly the player feedback loop between video output and player input, makes game video streaming much more sensitive to latency than traditional video streaming. Existing video coding methods can trade computational power, and little else, for reductions in encoding time. New methods for integrating the encoding process into the video rendering process can provide significant reductions in encoding time while also reducing computational power, improving the quality of the encoded video, and retaining the original bitstream data format to preserve interoperability of existing hardware devices.
When a video game instance is running on hardware local to the player, it is desirable to have the game output each pixel at the highest quality. However, in a server-side game instance where rendered output is encoded and transmitted to a remote client, the encoder may reduce image quality to fit within a limited bandwidth. If rendered quality is dramatically higher than the quality of the decoded output, there is a measurable amount of server-side rendering work that is lost.
By adapting the server-side rendered quality to match the post-quantization quality based on feedback from the encoder, the game can reduce wasted server-side computation without any noticeable client-side quality loss. The reduction in server-side computational waste may also result in additional benefits including reduced energy usage, reduced rendering times, and reduced player-feedback latency. The server-side computational savings is compounded in environments where multiple game instances are running on the same server.
In streaming environments for games that involve multiple players, particularly games such as Massive Multiplayer Online Games (“MMOGs”), ensuring that server-side rendering work is not wasted becomes increasingly important. Due to the limited bandwidth available to players of MMOGs, an encoder that maximizes rendering quality while preventing a slowdown in the game is particularly important. Current technologies, as discussed below, adopt various methods to attempt to address this problem, but remain deficient.
U.S. Patent Publication No. US20170132830A1 (“the '830 Publication”), discloses systems and methods for determining a select shading point in a 3D scene on which shading is to be performed, performing the shading on the determined shading point, and determining shading information of the 3D scene based on a result of the shading performed on the determined shading point. The shading of the scene is adjusted based on temporal characteristics of the scene. However, this technology does not address the fundamental problem of optimizing encoding based on server-side rendering capabilities and available bandwidth.
U.S. Patent Publication No. US20170200253A1 (“the '253 Publication”) discloses systems and methods for improving rendering performance of graphics processors. At the graphics processor, an upper threshold can be set so that when a frame greater than the set threshold is encountered, the graphics processor takes appropriate action to reduce rendering time. However, this technology is based solely on a set threshold and does not dynamically adjust to server-side rendering capabilities and available bandwidth.
U.S. Patent Publication No. US2017/0278296A1 (“the '296 Publication”) discloses systems and methods in which the initial rendering of a scene that determines texture at each portion of the scene is generated, and a ray traced rendering of the scene is generated by tracing an initial sample of rays. This reference discloses that an optimal number of samples for each pixel is intelligently determined based on foreknowledge of scene textures and identifying noise arising due to under-sampling during ray tracing. Once more, this technology is limited to optimal ray sampling and does not dynamically adjust to server-side rendering capabilities and available bandwidth.
As is apparent from the above discussion of the state of the art in this technology, there is a need in the art for an improvement to the present computer technology related to the rendering and encoding of games.